Hand-eye calibration based on nonlinear optimization algorithm can not guarantee the convergence of the objective function to the global minimum, when there are errors in both robot forward kinematics and camera external parameters calibration. To solve this tricky problem, a new hand-eye calibration algorithm based on quaternion theory by convex relaxation global optimization was proposed. The critical factor of the angle between different interstation rotation axes by a manipulator was considered, an optimal set of relative movements from calibration data was selected by Random Sample Consensus (RANSAC) approach. Then, rotation matrix was parameterized by a quaternion, polynomial geometric error objective function and constraints were established based on Linear Matrix Inequality (LMI) convex relaxation global optimization algorithm, and the hand-eye transformation matrix could be solved for global optimum. Experimental validation on real data was provided. Compared with the classical quaternion nonlinear optimization algorithm, the proposed algorithm can get global optimal solution, the geometric mean error of hand-eye transformation matrix is no more than 1.4 mm, and the standard deviation is less than 0.16 mm.